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SIGNAL PROCESSING APPARATUS. METHOD OF THE SAME , 
AN IMAGE PROCESSING APPARATUS AND METHOD OF THE SAME 

BACKGROUND OF THE INVENTION 
1. Field of the Invention 



processing apparatus and method using a difference in 
sampling phase for removing aliasing components and to 
thereby broaden the frequency band of a signal and an 



image signal obtained by capturing a plurality of images 
having a predetermined difference in sampling phase by a 
single imaging element. 



processing method in a charged coupled device (CCD) or 
other image sensor for obtaining a broad band image 
doubled in the number of pixels by inputting two Images 
shifted by a pitch of exactly half of the sampling 
interval and up- sampling and adding the two. The signal 
processing method is applicable to a black and white CCD 
image sensor, triple -element RGB CCD image sensor, etc. 

Also, there have been attempts to apply this to 
an image obtained by a single -element CCD image sensor 



The present invention relates to a signal 



image processing apparatus and method for processing an 



2 . 



Description of the Related Art 



In the related art, there has been a signal 



having a color filter, currently the mainstream in video 
cameras and still image pickup devices (hereinafter also 
referred to as electronic still cameras). Since a single- 
element CCD image sensor has a color filter cycle of, for 
example, every two pixels etc. , it has been considered to 
shift images by half of the cycle, that is, one pixel. A 
signal of an image captured by a single -element CCD image 
sensor and converted from an analog to digital format is 
processed in a later camera signal processor to generate 
an RGB signal or a luminance signal and a color 
difference signal using interpolation for each color or 
an interrelationship among the colors . 

Summarizing the problem to be solved by the 
invention, in a single-element CCD image sensor, however, 
when shifting the pixels, the array of signals of the 
colors becomes different from the conventional array due 
to the increase in the number of pixels . The later camera 
signal processor therefore not only has to process an 
Increased number of pixels, but also has to change the 
algorithm itself. 

Also, when shifting the pixels by less than the 
pixel pitch of the CCD image sensor, there is the 
disadvantage that the camera signal processor has to 
process a signal with uneven sampling, so the processing 
becomes complex. 
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For example, when using a Bayer array single- 
element CCD Image sensor obtaining a R (red), G (green), 
and B (blue) pixel array pattern shown in Fig. 12A to 
obtain a total of four Images shifted one pixel each in 
5 the vertical, horizontal, and diagonal directions, the 

obtained pixel arrays of the R, G, B data become as shown 
in Figs. 12E, 12F, and 12G, which are different from the 
normal pixel arrays shown in Figs. 12B, 12C, and 12D. 
01 Namely, as will be understood by comparing 

10 Figs. 12C and 12D with Figs. 12F and 12G, the numbers of 
samples of R data and B data are increased four times, 
while as is understood by comparing Fig. 12B and Fig. 
12E, the number of samples of G data is only doubled. 

Therefore, the camera signal processor has to 
15 process R, G, B data of the pixel arrays shown in Figs. 

12E, 12F, and 12G, so there is the disadvantage that the 
processing becomes complex. 

Also, when shifting an image by less than the 
pixel pitch, the sampling Intervals become uneven, so 
20 there is the disadvantage that the processing becomes 
difficult. 



SUMMARY OF THE INVENTION 

An object of the present invention is to provide an 
2 5 image processing apparatus and method capable of 
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obtaining a high resolution Image from a low resolution 
Image without making the processing in the camera signal 
processor complex and a signal processing apparatus and 
method using the band broadening method used in the image 
5 processing apparatus etc. 

To attain the above object, according to a first 
aspect of the present invention, there is provided a 
signal processing apparatus comprising a transforming 
means for transforming a plurality of first digital 

10 signals having mutually different sampling phases to 
generate a plurality of second digital signals in 
frequency domain; a memory means for storing a plurality 
of complex numbers corresponding to the sampling phases; 
and a processing means for multiplying the plurality of 

15 second digital signals and the plurality of complex 

numbers corresponding to the plurality of second digital 
signals and adding the multiplication results to generate 
a third digital signal free from aliasing components. 

In the signal processing apparatus of a first aspect 

20 of the present invention, the transforming means 

transforms the plurality of first digital signals having 
mutually different sampling phases a plurality of second 
digital signals in the frequency domain. 

Then, the processing means multiplies the plurality 

2 5 of second digital signals with a plurality of complex 
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number's stored In the memory means corresponding to the 
plurality of second digital signals and adds the 
multiplied results to generate third digital signals 
freed from aliasing components. 
5 Preferably, the apparatus further comprises a phase 

shift means for shifting the phase of the first digital 
signals or the second digital signals by exactly a 
predetermined phase. 

According to a second aspect of the present 

10 invention, there is provided a signal processing 

apparatus, comprising a phase shift means for shifting 
the phase of a plurality of first digital signals having 
mutually different sampling phases to generate second 
digital signals; a memory means for storing a plurality 

15 of real numbers indicating real parts and Imaginary parts 
of a plurality of predetermined complex numbers 
corresponding to the sampling phases; a processing means 
for multiplying the first digital signals with real 
numbers indicating the real parts corresponding to the 

20 first digital signals to obtain first multiplied results, 
multiplying the second digital signals corresponding to 
the first digital signals with real numbers indicating 
the imaginary parts corresponding to the second digital 
signals to obtain second multiplied results, and adding 

2 5 the first multiplied results and the second multiplied 
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results to generate third digital signals free of 
aliasing components . 

According to a third aspect of the present 
invention, there is provided an image processing 
5 apparatus, comprising an image input means for generating 
a plurality of first image signals having mutually 
different sampling phases in accordance with Imaging 
results; a transforming means for transforming the first 
image signals to a frequency domain to generate a 

10 plurality of second image signals; a memory means for 

storing a plurality of complex numbers corresponding to 
the sampling phases; a processing means for multiplying 
the plurality of second image signals with the plurality 
of complex numbers corresponding to the plurality of 

15 second Image signals and adding the multiplied results to 
generate third image signals free from aliasing 
components . 

Preferably, the image input means forms an image of 
a plurality of color lights passed through a single-plate 

20 type color filter on corresponding pixels among a 
plurality of pixels arranged in a matrix two- 
dimensionally to generate the first image signals 
comprised by color data of the plurality of colors; and 
the processing means performs the multiplication and the 

2 5 addition for every color data of the plurality of color 
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data to generate a plurality of fourth Image signals 
corresponding to the plurality of colors and generates 
the third image signals by using the plurality of fourth 
image signals. 

5 More preferably, the sampling phase is determined 

for a predetermined one color among the plurality of 
colors in order that a sampling pattern of color data of 
the color Included in the plurality of first image 
signals and a sampling pattern of color data included in 

10 the fourth image signals of the color become similar. 

Preferably, the processing means comprises a spatial 
shift means for spatially shifting the second image 
signals in accordance with the sampling phases; a basic 
spectrum calculation means for multiplying the spatially 

15 shifted plurality of second image signals with the 
plurality of complex numbers corresponding to the 
plurality of second image signals and adding the 
multiplied results to calculate a basic spectrum free 
from aliasing components; and an Inverse transforming 

20 means for transforming the basic spectrum from a 

frequency domain to a time domain to generate the third 
image signals. 

Alternatively, preferably the apparatus further 
comprises a drive means for moving the image input means 

25 physically, optically, or electrically so that the image 
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Input means can generate a plurality of Image signals 
having mutually different sampling phases in accordance 
with the imaging results. 

Alternatively, more preferably the image input means 
5 is a single-element CCD image sensor and the color filter 
is a primary color filter or a color compensation filter. 

According to a fourth aspect of the present 
invention, there is provided an image processing 
apparatus , comprising an image input means for receiving 

10 as input a plurality of first image signals having 

mutually different sampling phases in accordance with 
imaging results; a phase shift means for shifting the 
phase of the plurality of first digital signals to 
generate second digital signals; a memory means for 

15 storing a plurality of real numbers respectively 

indicating real parts and imaginary parts of a plurality 
of predetermined complex numbers corresponding to the 
sampling phases; a processing means for multiplying the 
first digital signals with real numbers Indicating the 

20 real parts corresponding to the first digital signals to 
obtain first multiplication results, multiplying the 
second digital signals corresponding to the first digital 
signals with real numbers indicating the imaginary parts 
corresponding to the second digital signals to obtain 

2 5 second multiplied results, and adding the first 



multiplied results and tlie second multiplied results to 
generate third digital signals free from aliasing 
components. 

According to a fifth aspect of the present 
invention, there is provided a signal processing method 
comprising transforming a plurality of first digital 
signals having mutually different sampling phases to a 
frequency domain to generate a plurality of second 
digital signals; multiplying the plurality of second 
digital signals with a plurality of complex numbers 
corresponding to the plurality of second digital signal 
and adding the multiplied results to generate third 
digital signals free from aliasing components. 

According to a sixth aspect of the present 
invention, there is provided a signal processing method 
using a plurality of real numbers indicating real parts 
and imaginary parts of a predetermined plurality of 
complex numbers corresponding to sampling phases, 
comprising shifting a plurality of first digital signal 
having mutually different sampling phases by 
predetermined phases to generate second digital signals 
multiplying the first digital signals with real numbers 
indicating the real parts corresponding to the first 
digital signals to generate first multiplied results; 
multiplying the second digital signals corresponding to 
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the first digital signals with real numbers indicating 
the imaginary parts corresponding to the second digital 
signal to generate second multiplied results; and adding 
the first multiplied results and the second multiplied 
5 results to generate third digital signals free of 
aliasing components . 

According to a seventh aspect of the present 
invention, there is provided an image processing method 
comprising generating a plurality of first image signals 

10 having mutually different sampling phases in accordance 
with imaging results; converting the first image signals 
to a frequency domain to generate a plurality of second 
image signals; and multiplying the plurality of second 
image signals with a plurality of complex numbers 

15 corresponding to the plurality of second image signals 
and adding the multiplication results to generate third 
image signals free from aliasing components. 

Preferably, the method further comprises forming an 
image of a plurality of color lights passing through a 

20 single-plate type color filter on corresponding pixels 
among a plurality of pixels arranged in a matrix two- 
dimensionally to generate the first image signals 
comprised by color data of the plurality of colors and 
performing the multiplication and the addition for every 

2 5 color data of the plurality of colors to generate a 



plurality of fourth Image signals corresponding to the 
plurality of colors and generate the third image signals 
by using the plurality of fourth image signals. 

More preferably, the method further comprises 
determining the sampling phase for a predetermined one 
color among the plurality of colors so that a sampling 
pattern of color data of the color included in the 
plurality of first image signals and a sampling pattern 
of color data included in the fourth image signals of the 
color become similar. 

According to an eighth aspect of the present 
invention, there is provided an image processing method 
using a plurality of real numbers prepared in advance 
indicating real parts and imaginary parts of a 
predetermined plurality of complex numbers corresponding 
to sampling phases, comprising generating a plurality of 
first image signals having mutually different sampling 
phases in accordance with imaging results; shifting by a 
predetermined phase the plurality of first digital 
signals to generate second digital signals; multiplying 
the first digital signals with real numbers indicating 
the real parts corresponding to the first digital signals 
to generate first multiplied results; multiplying the 
second digital signals corresponding to the first digital 
signals with real numbers indicating the imaginary parts 
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corresponding to the second digital signals to generate 
second multiplied results; and adding the first 
multiplied results and the second multiplied results to 
generate third digital signals free of aliasing 
components . 

BRIEF DESCRIPTION OF THE DRAWINGS 

These and other objects and features of the present 
invention will become clearer from the following 
description of the preferred embodiments given with 
reference to the accompanying drawings, in which: 

- " " /-lew- for explaining an RGB Bayer array 




still camera of a first embodiment of the present 
invention; 

Fig. 2 is a view of a part of the configuration of 
the electronic still camera of the first embodiment of 
the present invention; 

Fig. 3 is a view for explaining a sampling phase of 
four Images captured by a CCD image sensor shown in Fig. 
2 for obtaining one high resolution image; 

' ^Fl-g- — 4 — ic- a vie w- for explaining processing in a 
signal processor shown in Fig. 2; 

Fig. 5A is a view for explaining a sampling phase of 
R data to be processed in a signal processor shown in 
Fig. 2 and Fig. 5B is a view for explaining a sampling 



phase of B data to be processed in the signal processor 
shown in Fig. 2; 

Fig. 6 is a view of the configuration of the signal 
processor shown in Fig. 2; 

Fig. 7 is a view of a part of the configuration of 
an electronic still camera of a second embodiment of the 
present invention ; 

Fig. 8 is a view of the configuration of a signal 
processor shown in Fig. 7; 

Fig. 9 is a view of a part of the configuration of 
an electronic still camera of a third embodiment of the 
present invention ; 

^F4-g 1 » _1 p * tt-i for explaining a color 
compensating filter used in the CCD image sensor shown in 
Fig. 9; 

Fig. 11 is a view of the configuration of a signal 
processor shown in Fig. 9; and 

-Fig-. — 1 2 1c -a-^view- for explaining disadvantages of 
the related art. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Below, an electronic still camera according to the 
present embodiments will be explained. 
First Embodiment 

An electronic still camera of the present embodiment 



is provided with an RGB Bayer array single -mode CCD image 
sensor and processes four sets of RGB data different in 
sampling phases obtained by the CCD image sensor using a 
predetermined band broadening method to increase the 
resolution and thereby obtain RGB data having a double 
bandwidth corresponding to double the number of pixels 
vertically and horizontally. 

Specifically, the electronic still camera obtains G 
data, R data, and B data of the pixel arrays shown in 
Figs. IB, 1C, and ID from the four sets of RGB data 
different in sampling phase. Including the RGB data of 
the pixel array shown in Fig. 1A, and add these to obtain 
the RGB data of the pixel array shown in Fig. IE. 

Figure 2 is a view of a part of the configuration of 
an electronic still camera of the present embodiment - 

As shown in Fig. 2, the electronic still camera 1 
comprises a lens 2, an optical LPF 3, a CCD image sensor 
4, an AD converter 5, a CCD drive 6, a memory 7, a signal 
processor 8, and a camera signal processor 9. 

[Optical LPF 3] 

The optical LPF 3 is an optical low pass filter 
(LPF) for removing unnecessary high band components from 
light from the object being photographed passing through 
the lens 2 . 

[CCD 4] 
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The CCD image sensor 4 comprises a plurality of 
photodiodes arranged in a matrix and a Bayer color filter 
array. The Bayer color filter array is comprised of G 
(green) color filters arranged in a checkerboard pattern 
5 and R (red) filters and B (blue) filters arranged in a 
checkerboard pattern in the remaining parts. The 
photodiodes receive light passing through the Bayer color 
filter array and convert the light to electrical signals 
to generate received light signals S4 0 to S4 3 4 in 
10 accordance with the amount of received light and outputs 
the received light signals S4 0 to S4 3 to the AD 
converter , 

The CCD image sensor 4 is, as will be explained 
later on, driven to move by the CCD drive 6 and captures 
15 four images different in sampling phases for obtaining a 
single high resolution image. 
[ AD Convert er 5 ] 

The AD converter 5 converts the received light 
signals S4 0 to S4 3 to image signals S5 0 to S5 3 and outputs 
20 an image signal S5 to the signal processor 8. 
[CCD Drive 6] 

The CCD drive 6 physically moves the CCD image 
sensor 4 by exactly a predetermined distance in a 
predetermined direction based on a control signal S8a 
2 5 from the signal processor 8 so that the CCD image sensor 
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4 captures a total of four Images respectively having 
phases of (0, 0), (n/2, -n/2), (0, n), (n/2, n/2) when 
assuming sampling intervals of the CCD image sensor 4 to 
be 2n and the phase of one image to be the origin (0, 0) 
5 as shown in Fig. 3. 

The signal processor 8 uses the image signal S5 of 
the four captured Images having mutually different 
sampling phases as shown in Fig. 3 input from the AD 
converter 5 to generate an image signal S8b of an image 

10 having double the number of pixels in the vertical and 
horizontal directions. 

Here, the image signal S5 is composed of the pixel 
data of pixels arranged in the matrix constituting the 
CCD image sensor 4. Pixel data of pixels at positions 

15 indicated by "G" in Fig. 4A Indicate G (green) data, 

pixel data of pixels at the positions indicated by "R" in 
Fig. 4B Indicate R (red) data, and pixel data of pixels 
at the positions indicated by "B" in Fig. 4C indicate B 
(blue) data. 

20 The signal processor 8 writes the image signal S5 of 

the four captured Images input from the AD converter 5 in 
the memory 7 , then reads It out and processes the image 
signal S5 by a later explained band broadening method to 
generate an image signal S8b so as to generate R data and 

2 5 B data of the pixel array patterns shown in Figs. 1C and 
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ID from R data and B data of the pixel array patterns 
shown in Figs, 4B and 4C. 

Note that the pixel array pattern of G data shown in 
Fig. 4A and the pixel array pattern of G data shown in 
5 Fig. IB are the same, so no processing is necessary for 
the G data. Namely, the signal processor 8 uses the G 
data included in the image signal S5 as it is as the G 
data of the image signal S8b. 

Also, the signal processor 8 generates a control 

10 signal S8a based on information on the direction of 

movement and distance of the CCD image sensor 4 stored in 
the memory 7 in advance so as to obtain four captured 
Images having mutually different sampling phase shown in 
Fig. 3 and outputs the control signal S8a to the CCD 

1 5 drive 6 . 

Below, the processings for generating R data and B 
data of the image signal S8b in the signal processor 8 
will be explained. 

Note that the processings for generating R data and 
20 B data are the same except for the value of the sampling 
phase . 

Here, the "sampling phase" Indicates the coordinates 
of a pixel corresponding to pixel data of a certain color 
included in the image signal S5 of the four captured 
2 5 images when assuming the position of a certain pixel 
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corresponding to pixel data of that color included in the 
image signal S8b to be the origin. 

Specifically, for the R data, the sampling phases of 
the four captured Images, as shown in Fig. 5A, 
respectively become (a x0 - a y0 ) = (n/2, 0), (a xl - a yl ) = (n, - 
n/2), (a x2 - a y2 ) = (n/2, n), and (a x3 - a y3 ) = (n, n/2). 

Also, for the B data, the sampling phases of the 
four captured Images, as shown in Fig. 5B, respectively 
become (a x0 - a y0 ) = (0, n/2), (a xl - a yl ) = (n/2, 0), (a x2 - a y2 ) = 
(0, 3n/2), and (a x3 * a y3 ) = (n/2, n). 

The values of these sampling phases are used as 
(a x0 - a y0 ) to (a x3 * a y3 ) at the time of the later explained 
spatial shifting and are stored in the memory 7 shown in 
Fig. 2 in advance. 

Figure 6 is a view of the configuration of the 
signal processor 8. 

As shown in Fig. 6, the signal processor 8 comprises 
an Interpolation circuit 50, a Fourier transform circuit 
51, a spatial shift circuit 52, a basic spectrum 
calculation circuit 53, and an inverse Fourier transform 
circuit 54. 

The Interpolation circuit 50 receives as inputs 
image signals S5 0 to S5 3 of the four captured images to 
obtain one high resolution image. Interpolates these and 
up- samples them two-fold to generate image signals S50 0 
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to S50 3 , and outputs the Image signals S50 0 to S50 3 to the 
Fourier transform circuit 51. 

Specifically, the interpolation circuit 50 inserts 
(M^-l) number of "O's" between the sampling positions in 
5 the x direction and (My-1) number of n 0's n between the 
sampling positions in the y direction for the image 
signals S5 0 to S5 3 to generate the image signals S50 0 to 
S50 3 . 

Note that and My are multiples indicating how 
10 many multiples of frequency components for restoration to 
in the x direction and y direction with respect to the 
Nyqulst frequency. In the present embodiment, both are 
"2". 

The Fourier transform circuit 51 converts the Image 
15 signals S50 0 to S50 3 input from the Interpolation circuit 
50 to generate image signals S51 0 to S51 3 of a frequency 
domain expression and outputs the Image signals S51 0 to 
S51 3 to the spatial shift circuit 52. 

The spatial shift circuit 52 performs two- 
20 dimensional spatial shifting on the R data and B data 
included in the image signals S51 0 to S51 3 obtained by 
imaging to generate image signals Y 0 to Y 3 to express 
that sampling was performed in the positions of Figs. 4B 
and 4C. 

25 Namely, the spatial shift circuit 52 multiplies R 
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data and B data of the image signals S51 0 to S51 3 with 
exp(-j(co x Qf x0 +co y a y0 )/(2n)) to exp(-j(co x a x3 +co y a y3 )/(2n)), 
respectively, to generate image signals Y 0 to Y 3 . 

Note that, at this time, the portion above the 
5 Nyquist frequency indicates negative frequencies both in 
the x direction and y direction. The cox or coy in this 
portion has to be made (co x -co 8X ) or (co y -co sy ) . Here, co BX and 
co sy respectively are sampling frequencies in the x 
direction and y direction before up- sampling. 

10 The basic spectrum calculation circuit 53 uses the 

image signals Y 0 to Y 3 input from the spatial shift 
circuit 52 and complex numbers w 0 to w 3 , w 0 ' to w 3 ' stored 
in the memory 7 shown in Fig. 2 to separately generate 
the two-dimensional image signals X 0 0 of the R data and B 

15 data and outputs the image signal S53 comprised of the 

generated image signals X 0#0 of the R data and B data and 
the G data of a pixel in the position shown In Fig. 4A to 
the inverse Fourier transform circuit 54. 



20 data and B data generated as explained later, the complex 
numbers w 0 to w 3 to be multiplied with the image signals 
Y 0 to Y 3 for finding the basic spectrum components X13 0 0 
in the first and third quadrants and the complex numbers 
w 0 ' to w 3 ' to be multiplied with the image signals Y 0 to 

2 5 Y 3 for finding the basic spectrum components X24 0 0 of the 



Note that the memory 7 stores, for each of the R 



second and fourth quadrants. 

The basic spectrum calculation circuit 53 obtains 
basic spectrum components X13 0#0 not containing aliasing 
components based on the formula (1) below and obtains 
basic spectrum components X24 0#0 not containing aliasing 
components based on the formula (2) below: 

X13 0 o = -{(1- j)Y 0 +(1+ J)Y ± +(1- j)Y 2 +(1+ j)Y 3 | 

...(1) 

Here, ( 1- J ) , ( 1 + j ) , ( 1-J ) and (1 + J) in the formula (1) 
are defined as w 0 , w L , w 2 and w 3 respectively, 

X24 Q 0 = ^{(1 + 3)Y 0 +(1- j)Y 1+ (l + j)Y 2 +(l- j)Y 3 } 

...(2) 

Here, ( 1+ J ) , ( 1- j ) , ( 1 + j ) and (1-J) in the formula (2) 
are defined as w 0 - , Wj/ , w 2 - and w 3 - respectively. 

Then, as shown in formula (3) below, the basic 

spectrum components X13 00 and X24 0 0 are added to generate 

the basic spectrum component X 0 0 . 

= X13 rt + X24 rt rt ...(3) 
0,0 0,0 0,0 

Below, the grounds of the above formulas (1) and (2) 

will be explained. 

First , the theory behind the band broadening method 



4 



us 
J* 
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using a plurality of discrete signals having differences 
In sampling phase in a two-dimensional direction 
constituting the grounds of the above formulas (1) and 
(2) will be explained. 
5 The variables are defined below: 

S total number of input digital signals (discrete 
signals ) 

L indexes 0, 1, S-l of digital signals 

Y_org L L-th organization digital signal 

t jl 10 (frequency domain expression) 

■j\ y_org L L-th organization digital signal (spatial 

Cj domain expression) 

El Y L L-th organization digital signal spatially 

Li 

SI shifted (frequency domain expression) 

M 

15 y£, L-th organization digital signal spatially 

shifted (spatial domain expression) 
a xL sampling phase difference (rad) in the x-axis 
direction with respect to 0th organization 
signal of L-th organization digital signal 

20 a yL sampling phase difference (rad) in the y-axis 

direction with respect to 0th organization 
signal of L-th organization digital signal 
X ± k imaging components X( Go x -i-co 8x , co y -l*co sy ) of i- 

order in the x-direotlon and k-order in the y- 

2 5 direction, basic spectrum when i=k=0 
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P x number of imaging components having negative 

order in tlie x-direotion 
Q x number of imaging components having positive 
order in the x-direction 
5 P y number of imaging components having negative 

order in the y- direction 
Q y number of imaging components having positive 
£f order in the y-direction 

S| 3 

-j N x multiple Indicating how many times of 

10 components with respect to Nyquist frequency 

til were included in continuous signal input before 

s 

O being sampled in x-direotion 

- 

U multiple indicating how many times of frequency 

p = 

CI components for restoration in the x-direotion 

1= I 

15 with respect to Nyquist frequency normally, 

N X =M X 

N y multiple indicating how many times of 

components with respect to Nyquist frequency 
were included in continuous signal input before 
20 being sampled in y-direotlon 

multiple indicating how many times of frequency 
components for restoration in the y-direction 
with respect to Nyquist frequency, normally, 

25 w L complex number for multiplying with L-th 



organization digital signal in frequency domain 



A signal with sampling phase shifted by exactly a x 
in the x- direction and a y in the y- direction with respect 
to a reference discrete signal is, when indicating the 
reference discrete signal by the formula (4) below, 
indicated by the formula (5) below. 

00 00 

Y(o) ,(o )= Y 2 X(<y -±G) o) -ka> ) ...(4) 
x' y ±=-oo k=-oo x sx y sy 

Y(a> x ,a>y ) = exp(j * (^ x a x + " a y ) / 2:7r ) * ( 

0O 00 

2 2 exp(-j-(ia x +ka y ))X(a; x -icw SXl cwy -ka; S y) 

j=-oo k = -oo 

...(5) 

Here, two-dimensional spatial shift can be attained 
by shifting to a correct position after interpolation 
when in the spatial domain. 

Also, when In the frequency domain, it is sufficient 
to multiply with "exp( - j (cox ax+coy ay ) / ( 2n) ) " . 

The spatial shift circuit can be realized for both 
the spatial domain and frequency domain. 

For example, when performing two-dimensional spatial 
shift in the spatial domain, in Fig. 6, the Fourier 
transform circuit 51 and the spatial shift circuit 52 are 
switched. Namely, the Fourier transform is performed 
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after spatial shift. 

Next, consider a complex number simultaneous 
equation for obtaining a complex number w L to be 
multiplied wltb S number of discrete signals to reproduce 
5 basic spectrum components . 

When in one dimension, the fact that the spectrum 
becomes symmetric between a positive frequency and a 
negative frequency based on the characteristics of a 
Fourier transform is used. It is sufficient to consider 

10 only a positive frequency and the number of imaging 
components in the positive range. 

On the other hand, as in the present embodiment, 
when in two dimensions, due to the characteristics of a 
Fourier transform, conjugation occurs about the origin 

15 and not about the cox- axis or coy- axis. Namely, on a cox- coy 
plane, the first and third quadrants are conjugated and 
the second and the fourth quadrants are conjugated. 
Namely, the two quadrants next to each other, for 
example, the first quadrant and the second quadrant, are 

20 independent from each other. Accordingly, It is necessary 
to consider the imaging components included in two 
quadrants, for example, the first and second quadrants. 
Note that the remaining two quadrants can be obtained by 
using the conjugated relationship. 

25 Thus, let us consider the first and the second 
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quadrants. First, consider the first quadrant. The number 
of discrete signals S necessary for obtaining solutions 
of a complex number simultaneous equation of the first 
quadrant becomes the product of the number of discrete 
5 signals necessary in the x-directlon and the number of 
discrete signals necessary in the y- direction, that is, 
S= ( P x +Q x + 1 ) ( P y +Q y + 1 ) - 

Ci Here, 

41 

01 P x =FLoor ( ( N x - 1 ) / 2 ) 

j\ 10 Q x =FLoor((N x +M x -l)/2) 

Jj P y =FLoor((N y -l)/2) 

p Q y =FLoor ( (N y +My-l)/2) 

Note that in the present embodiment, when N x , N y , 

ht's 

H and My are all "2", S becomes "4". If obtaining four 

?! 

15 organizations of continuous signals, the imaging 

components can be removed and a signal up double the 
Nyquist frequency can be obtained. 

Accordingly, the complex number simultaneous 
equation can be indicated as the formula (6) below when 

20 the number of formulas and the unknown number of the 

basic spectrum components X o0 and imaging components X ± K 
are S and the discrete signal of the first and the third 
quadrants after spatial shifting is assumed to be Y13 ± x . 
By solving the formula (6) below to obtain the complex 

2 5 number W L , the basic spectrum components of the first and 
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the third quadrants can be obtained as shown In the 
formula (7) below. Note that W L in the formula (7) below 

indicates a complex number. 

fix Qy 

y13 l= 2 ^2 es P(-J-( ia X L+ka yL ))X irk 

i Px,k k Py 

5 where, L=0 , 1, ...» S- 1 

... (6) 

CJ ^13o.o=2W^3, ...(7) 

U! ' = ° 

Si 
*= 

l? i Next , if considering the second quadrant , co x is 

« =■ 

4} 

a 10 negative in the second quadrant and the Imaging 

™ =: 
is? 

gi components in the x- direction also becomes a negative 

CI 

M order. Therefore, the complex number simultaneous 

CI 

O equation can be indicated as in the formula (9) below 

when assuming the number of formulas and the unknown 

15 number of basic spectrum components X 0#0 and imaging 

components X ± k to be S and assuming the discrete signals 
in the second and fourth quadrants after spatial shifting 
to be Y24 L . By solving the formula (8) below to obtain 
the complex number w L ', the basic spectrum components 

20 X24 0 0 can be obtained as shown In the formula (9) below. 

Note that w L ' of the formula (9) below indicates a 
complex number different from w L . 
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Px Qy 

i ^—Qx k -~-Py 
where L=0 , 1, - - - , S-l 

. . .(8) 

X24 00 = ^w L 'Y24 L ...(9) 

The complex number w L obtained from the above ±s 
used for removing the Imaging components in the first and 
third quadrants, while the complex number w L ' is used for 
removing the Imaging components in the second and the 
fourth quadrants. 

Here, since the processing for removing the imaging 
components in the first and the third quadrants and the 
processing for removing the imaging components of the 
second and the fourth quadrants are independent, the 
signals are divided into two systems . 

The processing for dividing the components into two 
systems generates a signal where the values of the second 
and fourth quadrants are replaced by "0" in the frequency 
domain and a signal where the values of the first and 
third quadrants are replaced by "0" in a frequency 
domain. On the other hand, when in a spatial domain, it 
generates a signal filtered for extracting the frequency 
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components of the second and the. fourth quadrants and a 
signal filtered for extracting frequency components of 
the first and third quadrants. 

Multiplying the complex numbers w L and w L ' 
5 respectively with the two organizations of signals gives 
the basic spectrum X13 L of the first and third quadrants 
wherein imaging components are removed and the basic 
spectrum S24 L of the second and fourth quadrants wherein 
Imaging components are removed, 
10 Finally, the basic spectrum X13 L and the basic 

spectrum X24 L from which the imaging components have been 
removed may be added to obtain the desired band broadened 
signal . 

This completes the explanation of the theory behind 
15 the band broadening method. 

Below, the grounds of the above formulas (1) and (2) 
will be explained based on the above theory of the band 
broadening method . 

Namely, a case where imaging components are removed 
20 from R data of pixels at positions shown in Fig. 4B, 

obtained from the image signal S5 of the four captured 
images , to broaden the band and R data of pixels at 
positions shown in Fig. 1C are generated will be 
explained. 

2 5 Note that the case where imaging components are 
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removed from B data of pixels at positions shown in Fig* 
4C, obtained from the image signal S5 of the four 
captured images, to broaden the band and B data of pixels 
at positions shown in Fig. ID are generated is the same 
5 as the case of R data after removal of values of the 
sampling phase differences a^. and a yL . 

In the first and third quadrants, the formulas (10a) 
□ to (lOd) are obtained from the above formula (6). 

a! 

m 

T 10 Y 0 =X 0 0 - jX 10 +X 01 - jX^i . . . (10a) 

ai 

Yx=X 0 0 -X x .o+JXo^-jX, tl . . .(10b) 

U Y 2 =X 0 , 0 - jX, rl . . . (10c) 

|| Y 3 =X 0 , 0 -X 1#0 -jX 0 ,! + JX lrl . . . (lOd) 



O 



15 When solving the above formulas (10a) to (lOd), the 

above formula (1) corresponding to the above formula (7) 
is Introduced and the basic spectrum X13 0 0 of the first 
and third quadrants is obtained. 

Regarding the second and fourth quadrants, the 

2 0 formulas (11a) to (lid) are obtained from the above 
formula ( 8 ) . 

Y 0 =X 0i0 +3X 1 , 0 +X 0 , 1 +JX 1 ml . . . ( 11a) 

Y 1= X 0 , 0 -X x .o+:JX 0 ^-JX, rl . . . (lib) 

25 Y 2 =X 0f0 +jX lt0 -X 0>1 -jX lfl . . . (11c) 
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Y 3 -X 0 ,0-Xi , 0 - jXo^+JX, tX ...{ lid) 

When solving the above formulas (11a) to (lid), the 
above formula (2) corresponding to the above formula (9) 
5 is derived and the basic spectrum X24 0#0 of the second and 
the fourth quadrants is obtained. 

The inverse Fourier transform circuit 54 performs 
inverse Fourier transform on the image signal S53 input 
from the basic spectrum calculation circuit 53 to obtain 
10 a band broadened digital signal S8 . 

The camera signal processor 9, taking account of the 
correlation of the colors , generates an interpolated RGB 
signal or a luminance signal and a color difference 
signal from R, G, B data included in the image signal S8 
15 from the signal processor 8. 

Below, the operation of an electronic still camera 
of the present embodiment shown in Fig. 2 will be 
explained . 

The CCD image sensor 4 moves in a predetermined 
20 direction by an exactly predetermined distance by being 
driven by the CCD drive 6 . As shown in Fig . 3 , it 
therefore captures four Images having mutually different 
sampling phases and outputs received light signals S4 0 to 
S4 3 based on the captured four Images to the AD converter 
25 5. 
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Next, the AD converter 5 converts the received light 
signals S4 0 to S4 3 to digital image signals S5 0 to S5 3 and 
outputs them to the signal processor 8 . The image signals 
S5 0 to S5 3 are stored in the memory 7, then read and 
5 processed in the signal processor 8. 

In the signal processor 8, first, the interpolation 
circuit 50 Interpolates the input image signals S5 0 to 
S5 3 and up -samples them two-fold to generate the image 
signals S50 0 to S50 3 . 

10 Then, the Fourier transform circuit 51 performs two- 

dimensional Fourier transform on the image signals S500 
to S503 to generate the image signals S510 to S513 of the 
frequency domain expression. 

Then, the spatial shift circuit 52 multiples the R 

15 data and B data of the image signals S510 to S513 with 
exp(-j(co x a x0 + Go y a y0 ) / ( 2n ) ) to exp(-j(co x a x3 + co y 
a y3 )/(2n)) to generate the image signals Y 0 to Y 3 - 

The basic spectrum calculation circuit 53 
uses the input Image signals Y 0 to Y 3 and the complex 

20 numbers stored in the memory 7 shown in Fig. 2 to 

separately generate an image signal S53 composed of a 
two-dimensional image signal X 0 0 of the R data and B data 
and outputs the image signal S53 comprised of the 
generated image signal X 0#0 of the R data and B data and 

2 5 the G data of the pixels at positions shown in Fig. 4A to 
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the Inverse Fourier transform circuit 54. 

The inverse Fourier transform circuit 54 performs an 
inverse Fourier transform on the input image signal S53 
to broaden the band and generate a digital signal S8b. 
5 The camera signal processor 9, taking account of the 

correlation of colors, processes the R, G, B data 
included in the image signal S8b to generate an 
interpolated RGB signal or a luminance signal and color 
difference signal • 
10 Then, it outputs an image based on the generated RGB 

signal or the luminance signal and color difference 
signal . 

As explained above, according to the electronic 
still camera of the present embodiment, it is possible to 

15 obtain a high definition image signal corresponding to 

double the numbers of pixels of the numbers of pixels of 
the CCD image sensor 4 . 

Also, according to the electronic still camera of 
the present embodiment, although the processing of the 

20 signal processor 8 becomes more complicated compared with 
that of the related art, as explained above, since it is 
not necessary to process the G data in the signal 
processor 8, the amount of data processing in the signal 
processor 8 can be reduced compared with a case where all 

2 5 the R, G and B data has to be processed. 
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Second Embodiment 

Figure 7 is a view of a part of the configuration of 
an electronic still camera of the present embodiment . 

As shown in Fig. 7, the electronic still camera of 
5 the present embodiment is the same as that of the above 

explained first embodiment except for the processing in a 
signal processor 68. 

Figure 8 is a view of the configuration of the 
signal processor 68 shown in Fig. 7. 
10 As shown in Fig. 8, the signal processor 68 

comprises an interpolation circuit 70, a spatial shift 
circuit 71, and a broad band signal generation circuit 
72. 

The Interpolation circuit 70 interpolates the image 
15 signals S5 0 to S5 3 of the four captured Images having 

different sampling phases shown in Fig. 3 input from the 
AD converter 5 shown in Fig. 7 to generate the image 
signals S70 0 to S70 3 . 

The spatial shift circuit 71 shifts the image 
20 signals S70 o to S70 3 input from the Interpolation circuit 
70 in the x-direotion and y-dlrection by exactly the 
amount of the phase difference to generate image signals 
Yo to Ya in order to express the fact that the input 
image signals S70 0 to S70 3 were sampled at the positions 
25 in Figs. 4B and 4C. 
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Specifically, the spatial shift circuit 71 removes 
some samples from the top of the image signals S70 0 to 
S70 3 Increased in the number of samples or adds some 
samples of appropriate values at the top to generate the 
5 signals y 0 to y 3 . 

The spatial shift circuit 71 outputs the signals y D 
to y 3 to the broad band signal generation circuit 72. 

The broad band signal generation circuit 72 divides 
the image signals y 0 to y 3 to components wherein both x 
10 and y have positive frequencies and components wherein 

one of x and y has a negative frequency and the other has 
the opposite positive frequency. Namely, it applies two- 
dimensional filtering for taking out frequencies wherein 
both x and y are positive and a two-dimensional filter 
15 for taking out frequencies wherein one of x and y has a 
negative frequency and the other has the opposite 
positive frequency. The signals obtained in this way are 
yl3 L and y24 L . 

The broad band signal generation circuit 72 applies 
2 0 two-dimensional Hilbert transform on the image signals 

yl3 L and y24 L . Namely, it shifts the phase by n/2(rad) by 
a not shown phase shifting means. The thus obtained 
signals are respectively called yl3 L ' and y24 L ' . 

Then, the broad band signal generation circuit 72 
2 5 uses real numbers indicating the real part and imaginary 
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part of the complex number w L read from the memory 7 for 
yl3 L and processes [Re(w L ) yl3 L +Im(w L ) yl3 L ' ] . In the same 
way. It uses the real numbers indicating the real part 
and imaginary part of the complex number w L ' read from 
5 the memory 7 for y24 L and processes [Re(w L ' )*y24 L +Im(w L ' )y 
24 L ' ] . 

Here, Re(x) Indicates the real part and Im(x) 
indicates the imaginary part . 

The broad band signal generation circuit 72 adds the 
10 results of [«e(w L )-Yl3 L +Im(w L )-yl3 L B ] and [Re(w L f ) y24 L +Im(w L 
f )y24 L '] to cancel (remove) the aliasing to broaden the 
band and obtain the image signal S68. 

Note that in the present embodiment, the real 
numbers indicating the real part and Imaginary part of 
15 the complex numbers wL and wL ' are stored in the memory 7 
in advance . 

The same effects can be obtained by the electronic 
still camera of the present embodiment as by the above 
explained electronic still camera of the first 
20 embodiment. 

Third Embodiment 

An electronic still camera of the present embodiment 
will be explained with reference to the case of using a 
CCD image sensor having a color compensating filter 
2 5 instead of a Bayer primary color filter array. 
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Figure 9 Is a view of a part of the configuration of 
an electronic still camera of the present embodiment . 

As shown in Fig. 9, the electronic still camera 1 
comprises a lens 2, an optical LPF 3, a CCD image sensor 
5 84, an AD converter 5, CCD drive 86, a memory 87, a 
signal processor 88, and a camera signal processor 9. 

[CCD image sensor 84] 

The CCD image sensor 84 comprises a plurality of 
photodiodes arranged in a matrix and a color compensation 

10 filter. The color compensation filter comprises filters 
transmitting Cy (cyan) color, M (magenta) color, Y 
(yellow), color and W (white) color arranged in the 
pattern shown in Fig. 10A. A photodiode receives light 
passing through the color compensation filter, converts 

15 the result from light to an electrical signal to generate 
received light signals S840 to S843 based on the amount 
of light received, and outputs the received light signals 
S84 to the AD converter 5. 

The CCD image sensor 84, as will be explained later 

20 on, is driven to move by the CCD drive 86, captures four 
images having mutually different sampling phases,, and 
generates received light signals S84 0 to S84 3 in order to 
obtain a high resolution image shown in Fig. 10B. At this 
time, the sampling phases of the four Images may be any 

25 phases. In the present embodiment, for example as shown 
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in Fig. IOC, they are (0, 0), (n/2, 0), (0, n/2), and 
(n/2, n/2). 

[ AD converter 85] 

The AD converter 85 converts the received light 
signals S84 0 to S84 3 to digital image signals S85 0 to S85 3 
and outputs the image signals S85 0 to S85 3 to the signal 
processor 88. 

[CCD drive 86] 

The CCD drive 86 physically moves the CCD image 
sensor 84 in a predetermined direction by an exactly 
predetermined distance based on a control signal S88a 
from the signal processor 88 so as to give a sampling 
interval of the CCD image sensor 84 of 2n and so that the 
CCD image sensor 84 captures a total of four images 
having the four different sampling phases shown in Fig. 
10C. 

[Signal processor 88] 

The signal processor 88 uses the image signals S850 
to S853 of the four captured Images having the mutually 
different sampling phases shown in Fig. 10C input from 
the AD converter 85 to generate an image signal S88b of 
an image wherein the numbers of pixels in the vertical 
and horizontal directions are doubled. 

Also, the signal processor 88 generates a control 
signal S88a based on information regarding a direction of 
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movement and distance of the CCD Image sensor 84 stored 
in the memory 87 in advance so that the four captured 
images having mutually different sampling phases shown in 
Fig- IOC can be obtained and outputs the control signal 
5 S88a to the CCD drive 86. 

The signal processor 88 writes the image signals 
S850 to S853 of the four captured images of the pixel 
array pattern shown in Fig. 10A input from the AD 
converter 85 to the memory 87, then reads them and 
10 performs the later explained band broadening to generate 
an image signal S88b of a pixel array pattern shown in 
Fig. 10B. 

Below, the processings for generating Cy data. Ye 
data, W data, and G data of the image signal S88b in the 
15 signal processor 88 will be explained. 

Note that the processings for generating the Cy 
data. Ye data, W data, and G data are the same except for 
the values of the sampling phases. 

Here, the sampling phase, as shown in Fig. 10C, has 
20 as its origin the position of a pixel outputting a color 
in accordance with the Cy data 

Specifically, for the Cy data, the sampling phases 
of the four captured images, as shown in Fig. 10C, 
respectively become (a x0 ' a y0 ) = (°» °)* (a x i' a yl ) = (n/2, 0), 
25 (a x2 - a y2 ) = (n/2, n/2) , and (a x3 - a y3 ) = (0, n/2). 
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For the Ye data, the sampling phases of the four 
captured Images respectively become (a x0 - a y0 ) = (n/2, 0), 
(a xl - a yl ) = (n, 0). (a x2 - a y2 ) = (n/2, 0), and (a x3 - a y3 ) = (n, 
n/2). 

5 For the W data, the sampling phases of the four 

captured Images respectively become (a x0 - or y0 ) = (0. n/2), 
(a xl ' a yl ) = (n/2, n/2), (a x2 - a y2 ) = (0, n/2), and (a x3 

C) a y3 ) = (n/2, n). 

J\ 

-js v 

m For the G data, the sampling phases of the four 

m 

4* 10 captured Images respectively become (a x0 - a y0 ) = (n/2, n/2), 

if | 

& (a xl , a yl ) = (n, n/2), (a x2 - a y2 ) = (n/2, n), and (a x3 - a y3 ) = (n, 

n) . 



The values of these sampling phases are used as 



U 

ss. s. 

y ' 

s 3 

P (a x0 - a y0 ) to (a x3 - a y3 ) at the time of the later explained 

P 

15 spatial shifting and are stored in the memory 87 shown in 
Fig. 9 in advance. 

Figure 11 is a view of the configuration of the 
signal processor 88. 

As shown In Fig. 11, the signal processor 88 
20 comprises an interpolation circuit 90, a Fourier 

transform circuit 91, a spatial shift circuit 92, a basic 
spectrum calculation circuit 93 and an inverse Fourier 
transform circuit 94. 

The interpolation circuit 90 receives as inputs 
25 image signals S85 0 to S85 3 , interpolates and up-samples 



these two-fold to generate Image signals S90 0 to S90 3 , 
and outputs the Image signals S90 0 to S90 3 to the Fourier 
transform circuit 91. 

The Fourier transform circuit 91 performs a two- 
5 dimensional Fourier transform on the image signals S90 0 
to S90 3 input from the interpolation circuit 90 to 
generate image signals S91 0 to S91 3 of the frequency 
domain expression and outputs the Image signals S91 0 to 
S91 3 to the spatial shift circuit 92. 

10 The spatial shift circuit 92 performs two- 

dimensional spatial shifting of the Cy data. Ye data, W 
data, and G data included In the image signals S91 0 to 
S91 3 obtained by the imaging to generate the image 
signals Y 0 to Y 3 in order to express that sampling was 

15 performed at the above explained positions (a x0 - a y0 ) to 

( C*x3 ' C*y3 ) • 

Namely, the spatial shift circuit 92 multiplies the 
Cy data. Ye data, W data, and G data of the image signals 
S91 0 to S91 3 with exp(-j(co x a x0 +co y a y0 )/(2n)) to exp(-j(co x 
2 0 « x3 +co y a y3 )/(2n)), respectively, to generate the image 
signals Y G to Y 3 . 

Note, in this case, the portion higher than the 
Nyquist frequency indicates negative frequencies both in 
the x-direction and y-direotlon. co x or co y in that portion 
25 have to be made (co x -co sx ) or (co y -a> sy ) . Here, co sx and co sy 



- 42 - 



respectively aire sampling frequencies in the x-direction 
and y- direction before up- sampling. 

The basic spectrum calculation circuit 93 uses the 
image signals Y 0 to Y 3 input from the spatial shift 
5 circuit 92 and complex numbers stored In the memory 87 
shown in Fig. 9 to separately generate two-dimensional 
image signals X 0 0 of the Cy data. Ye data, W data, and G 
data and outputs to the inverse Fourier transform circuit 
94 an image signal S93 comprised by the generated image 

10 signals X G 0 of the Cy data. Ye data, W data, and G data. 

Note that the memory 87 stores, for the Cy data. Ye 
data, W data, and G data generated as will be explained 
later on, complex numbers w 0 to w 3 to be multiplied with 
the image signals Y 0 to Y 3 for obtaining the basic 

15 spectrum components X13 0 0 in the first and third 

quadrants and complex numbers w 0 ' to w 3 ' to be multiplied 
with the image signals Y 0 to Y 3 for obtaining the basic 
spectrum components X24 0#0 of the second and fourth 
quadrants . 

20 The basic spectrum calculation circuit 93 obtains 

basic spectrum components X13 00 based on the formula (13) 
below obtained from the formulas (12a to 12d) below and 
obtains basic spectrum components X24 00 based on the 
formula (15) below obtained from the formulas (14a to 

25 14d) below- 
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Yo=Xo.o+X 1 .o+X 0 . 1 +X 1 . 1 ...(12a) 

Y^Xo.o-jX^o+Xo.i-jX^ • - - ( 12b) 

Y a =X 0>0 +X l .„- jX 0-1 + jX 1#1 . . . ( 12o) 

Y 3 =Xo.o-dX l . 0 -3X 0 . 1 -X 11 . . . (12d) 

5 

X13 0 0 ={- JY 0 +Y 1 +Y 2 + jY 3 )/2 . . . ( 13 ) 

CI Y 0 =X 00 +X 1 , 0 +X 01 +X 1>1 ...(14a) 

U! Y 1 -X 0ia +jX li0 +X 0il +jX l<l ...(14b) 

m 

4° 10 Y^Xo.o+X^o-jXo^-JX^! ...(14c) 

. f ~ 

=.£ I 

i] Y 3 =X 0 ; 0 +3X 1 , 0 -3X oa +X 1 tl . . .(14d) 

3 

f i 
-a ~ 

|j X24 0 0 ={Y 0 -jY 1 +jY 2 +3Y 3 }/2 ...(15) 

CI 

" f 15 Then, as shown in the formula (16) below, the basic 

spectrum components X13 0 0 and X24 0#0 are added to generate 
the basic spectrum components X 0 0 . 



X OiO =X13 OfO +X24 0f0 ... (16) 

20 

Also, the basic spectrum calculation circuit 93, in 
the same way as for the Cy data, obtains basic spectrum 
components X13 00 and basic spectrum components X24 00 for 
the Ye data, W data, and G data, and adds them to 
2 5 generate basic spectrum components X 0 0 . 
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The Inverse Fourier transform circuit 94 performs an 
inverse Fourier transform on the image signal S93 input 
from the basic spectrum calculation circuit 93 to broaden 
the band and obtain the digital signal S88b. 
5 The camera signal processor 89 processes the Cy 

data. Ye data, W data, and G data included in the image 
signal S88b from the signal processor 88 to generate an 
interpolated RGB signal included or generate a luminance 
signal and color difference signal taking into account 
10 the correlation of the colors. 

Next, the operation of the electronic still camera 
of the present embodiment shown in Fig. 9 will be 
explained. 

The CCD image sensor 84 is driven by the CCD drive 
15 86 to move in a predetermined direction by exactly a 
predetermined distance, captures four Images having 
mutually different sampling phase as shown in Fig. 10C, 
and outputs the received light signals S84 0 to S84 3 based 
on the four captured Images to the AD converter 85. 
20 Next, the AD converter 85 converts the received 

light signals S84 0 to S84 3 to digital image signals S85 D 
to S85 3 and output them to the signal processor 88. The 
image signals S85 0 to S85 3 are stored in the memory 87, 
then read and processed in the signal processor 88. 
25 In the signal processor 88, first the interpolation 



- 45 - 



circuit 90 interpolates and up- samples two-fold the input 
image signals S85 0 to S85 3 to generate the image signals 
S90 0 to S90 3 . 

Then, the Fourier transform circuit 91 applies a 
5 two-dimensional Fourier transform to the image signals 

S90 0 to S90 3 to generate image signals S91 0 to S91 3 of the 
frequency domain expression * 

Then, the spatial shift circuit 92 multiplies the Cy 
data. Ye data, W data, and G data of the image signals 
10 S91 0 to S91 3 with exp(-j(co x a x0 + co y a y0 )/(2n)) to exp(-J(co x 
a x3 + coy a y3 )/(2n)) to generate the image signals Y 0 to Y 3 . 

The basic spectrum calculation circuit 93 uses 
the input image signals Y 0 to Y 3 and the complex numbers 
stored in the memory 87 shown in Fig. 9 to separate 
15 generate two-dimensional image signals X 0 0 of the Cy 

data. Ye data, W data, and G data and outputs an image 
signal S93 comprised of the generated image signals X 0 0 
of the Cy data. Ye data, W data, and G data to the 
inversed Fourier transform circuit 94. 
2 0 The inverse Fourier transform circuit 94 applies an 

inverse Fourier transform to the input image signal S93 
to broaden the band and generate the digital signal S88b. 

The camera signal processor 89 processes the Cy 
data. Ye data, W data, and G data included in the image 
2 5 signal S88b to generate an interpolated RGB signal or 
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generate a luminance signal and color difference signal 
taking into account the correlation of the colors. 

Then, it outputs an image in accordance with the 
generated RGB signal or the luminance signal and color 
5 difference signal. 

As explained above, according to the electronic 
still camera of the present embodiment, even when using a 
CCD image sensor 84 having the above color compensation 
filter, a high definition image signal corresponding to 
10 double the number of pixels of the CCD Image sensor 84 
both in the vertical and horizontal directions can be 
obtained without complicated processing in the camera 
processor 89* 

The present invention is not limited to the above 
1 5 embodiment s . 

For example, the orders of processing in the signal 
processor 8 shown in Fig. 2, the signal processor 68 
shown in Fig, 7 and the signal processor 88 shown in Fig. 
9 are not specifically limited to the above. 
20 Also, the hardware configuration is not limited to 

that in Figs. 2, 7, and 9. 

Also, in the present invention, processing in a 
spatial domain and processing in a frequency domain can 
be mixed together. 
25 Furthermore, in the above embodiments, the CCD image 
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sensors 4 and 84 were physically moved to obtain four 
captured images having different sampling phase, however, 
a plurality of captured images having different sampling 
phases may be obtained by for example optically or 
5 electrically changing a light path by using a 

birefringent plate without physically moving the CCD 
image sensor. 

Also, the values of the sampling phases are not 
limited to those described in the embodiments. 

10 Also, a CCD image sensor was shown as an example of 

an imaging means in the above embodiments, however, a 
CMOS sensor etc. may also be used. 

Also, in the above embodiments, a case where four 
captured Images are used for obtaining a high resolution 

15 image (broadened band) doubled in the vertical and 

horizontal directions was shown as an example, but the 
increase in resolution is not particularly limited to 
two. Further, the increase in resolution may be different 
between the vertical direction and in the horizontal 

20 direction. Also, the band broadening may be not two 
dimensional, but also one dimensional. 

Also, in the above embodiments, four captured images 
were used in order to obtain a high resolution image, 
however, the number of captured images used is not 

2 5 limited to four since it depends on the multiple in each 
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directions . 

Also, the arrays of the colors of the color filters 
shown in Fig. 1A and Fig. 10A can be any arrays. 

The phase shift means used in the spatial domain is 
5 not limited to the Hilbert transform and the amount of 
phase shift is not limited to n/2. 

Summarizing the effect of the invention, as 
explained above, according to the signal processing 
apparatus and the method of the present invention, a 
10 signal can be broadened in band by simple processing. 

Also, according to the image processing apparatus 
and the method of the present Invention, a high 
resolution image can be obtained from a low resolution 
image by simple processing. 
15 While the Invention has been described with 

reference to specific embodiment chosen for purpose of 
illustration, it should be apparent that numerous 
modifications could be made thereto by those skilled in 
the art without departing from the basic concept and 
20 scope of the invention. 



